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Abstract 


This  paper  proposes  the  use  of  a  new  Systems  Readiness  Level  (SRL)  scale  for  managing 
system  development  and  for  making  effective  and  efficient  decisions  during  the  defense 
acquisition  process.  This  scale  incorporates  both  the  current  Technology  Readiness  Lev¬ 
el  (TRL)  of  the  Department  of  Defense  (DoD)  and  the  concept  of  an  Integration  Readi¬ 
ness  Level  (IRL)  developed  by  Stevens  Institute  of  Technology.  The  paper  describes  the 
foundations  for  the  SRL  and  how  it  is  formulated;  it  also  demonstrates  the  SRL’s  applica¬ 
tion  within  the  defense  acquisition  process  using  a  sample  case  with  notional  readiness 
values. 
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1.  Introduction 

In  1999,  the  United  States  (US)  General  Accounting  Office  (GAO)1  stated  that  there  were  few 
metrics  used  within  the  US  Department  of  Defense  (DoD)  to  gauge  the  impact  of  investments  or 
the  effectiveness  of  processes  used  to  develop  and  transition  technologies.  It  asserted  that 
additional  metrics  in  technology  transition  were  needed  (GAO,  1999).  In  2002,  in  a  testimony 
before  the  Subcommittee  on  Readiness  and  Management  Support,  Committee  on  Anned 
Services  of  the  US  Senate,  the  GAO  further  explained  DoD  challenges  in  implementing  best 
practices;  it  suggested  the  DoD  needed  to  enable  success  through  the  demonstration  of  value  and 
the  credibility  of  new  processes  through  the  use  of  metrics  (GAO,  2002). 

To  address  these  compounding  challenges,  in  1999,  the  DoD  began  implementing  the 
Technology  Readiness  Level  (TRL)  as  a  metric  to  assess  the  maturity  of  a  program’s 
technologies  before  its  system  development  begins  (DoD,  2005a;  2005b).  Additionally,  the  DoD 
made  constructive  changes  to  its  approaches  to  acquisition  that  would  address  these  issues  by 
2001:  (1)  assuring  a  weapon  systems’  technologies  are  demonstrated  to  a  high  level  of  maturity 
before  beginning  its  program  and  (2)  using  an  evolutionary  or  phased  approach  to  developing 
such  systems  (GAO,  2002). 

Even  with  the  implementation  of  new  processes  and  practices  within  DoD  acquisition,  the 
challenges  are  still  significant  (e.g.,  over  the  next  five  years,  the  DoD  plans  to  invest  an 
estimated  $900  billion  to  develop  and  procure  weapons  systems  at  a  pace  that  far  exceeds  the 
availability  of  resources  (GAO,  2008)). 

Consequently,  despite  the  utility  and  value  of  the  TRL  as  a  metric  for  determining  technology 
maturity  before  transitioning  into  a  system,  we  contend  that  TRLs  were  not  intended  to  address 
systems  integration  nor  to  indicate  that  the  technology  will  result  in  successful  development  of  a 
system  (Gove,  2007;  Mandelbaum,  2007;  2008).  As  Baines  (2004)  describes,  “the  wrong 
technology,  or  even  the  right  technology  poorly  implemented,  can  be  disastrous”  (p.  447. 
Therefore,  in  this  paper  we  will  build  upon  a  concept  originally  proposed  by  Sauser,  Verma, 
Ramirez-Marquez  and  Gove  (2006)  for  the  development  of  a  System  Readiness  Level  (SRL) 
scale  that  incorporates  the  maturity  level  of  the  critical  components  and  the  interoperability  of 
the  entire  system.  A  fundamental  argument  to  this  approach  is  that  the  metrics  for  the  coupling 
and  maturation  of  multiple  technologies  and  systems  have  been  shown  to  be  unresolved  issues  of 
strategic  relevance  (Nambisan,  2002;  Watts  &  Porter,  2003).  In  addition,  component-level 
considerations  relating  to  integration,  interoperability,  and  sustainment  become  equally  or  more 
important  from  a  systems  perspective  during  acquisition  (Sandbom,  Herald,  Houston  &  Singh, 
2003). 

The  SRL  we  will  describe  and  demonstrate  is  a  function  and  scale  that  incorporates  the  current 
TRL  scale  along  with  a  scale  of  integration.  The  combination  for  utilization  of  the  SRL  we 
contend  aids  in  making  strategic  decisions  during  defense  acquisition.  The  resultant  SRL  scale 


1  This  agency  became  the  US  Government  Accountability  Office  on  July  7,  2004. 
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can  provide  an  assessment  of  overall  system  development  and  can  identify  potential  areas  that 
require  further  work  to  facilitate  prioritization.  This  new  SRL  scale  of  system  maturity  can  be 
used  with  decision-making  tools  for  the  potential  acquisition  of  systems — which  involve  the 
dependency  and  interplay  among  performance,  availability  (reliability,  maintainability,  and 
supportability),  process  efficiency  (system  operations,  maintenance,  and  logistics  support),  and 
system  lifecycle  cost. 

2.  Theoretical  Foundation 

In  program  management,  resources  are  frequently  allocated  with  the  purpose  of  executing  tasks 
to  maintain  schedule  and  budget.  This  can  lead  to  an  assignment-type  program  scheduling 
problem  (Salewski,  Schirmer  &  Drexl,  1997)  when  the  ultimate  objective  of  any  program  is  to 
realize  a  product  (or  system)  to  satisfy  a  customer.  A  fundamental  challenge  to  resolving  this 
problem  is  that  when  attempting  to  meet  the  emergent  needs  of  the  warfighter,  program 
managers  (PMs)  will  often  continue  development  of  a  system  through  the  acquisition  lifecycle — 
while  they  coordinate  the  design  activities  with  preliminary,  ambiguous,  or  subjective 
information  (Pich,  Loch  &  De  Meyer,  2002).  The  balance  between  customer  needs  (e.g., 
warfighter)  and  design  activities  creates  a  tension  between  the  overview  required  by  the  program 
manager  and  the  detail  that  is  the  focus  of  the  system  developers  (de  Haes,  2006).  To  find  a 
concession,  organizations  have  relied  on  subjective  assessment  techniques  for  developing  the 
program  overview,  which  then  becomes  the  basis  for  making  strategic  acquisition  decisions. 
However,  these  subjective  assessments  are  human-intensive,  error-prone,  and  inadequate  for  the 
desired  management  controls;  such  controls  should  be  based  on  system  attributes  that  can  be 
quantitatively  measured  using  system  metrics  (Yacoub  &  Ammar,  2002).  The  tension  between 
subjectivity  and  detail  is  rationalized  through  prescriptive  techniques — which  allow  people  to 
make  better  decisions  by  using  normative  models,  but  with  knowledge  of  the  limitations  and 
descriptive  realities  of  human  judgment  (Smith  &  Winterfeldt,  2004). 

Within  agencies  of  the  US  government,  the  prescriptive  tool  and  soft  metric  of  the  TRL  has  been 
used  as  an  assessment  of  the  maturity  of  evolving  technologies  prior  to  incorporating  them  into  a 
system  or  sub-system.  The  original  TRL  was  a  bi-product  of  the  National  Aeronautics  and  Space 
Administration’s  (NASA)  post-Apollo  era  as  ontology  for  contracting  support  (Sadin,  Povinelli 
&  Rosen,  1989).  In  the  last  nine  years,  other  government  agencies  and  contractors  have  adopted 
the  TRL  scale  with  specific  variations  to  satisfy  their  needs  (e.g.,  the  Department  of  Defense 
(DoD),  the  Department  of  Energy  (DoE),  the  National  Air  and  Space  Intelligence  Center). 

There  have  been  many  attempts  to  identify  alternative  readiness/maturity  levels  that  will 
complement  the  TRL,  such  as  Design  Readiness  Level,  Manufacturing  Readiness  Level, 
Software  Readiness  Level,  Operational  Readiness  Level,  Human  Readiness  Level,  Habitation 
Readiness  Level  and  Capability  Readiness  Levels  (Bilbro,  2007;  Connelly,  Daues,  Howard  & 
Toups,  2006;  Cundiff,  2003).  Unfortunately,  each  has  faltered  in  addressing  the  core  issue  with 
the  TRL  as  identified  in  recent  literature;  thus,  the  legacy  constraints  with  the  TRL’s  abstraction 
have  remained.  These  constraints  are:  (1)  the  inability  to  represent  integration  between 
technologies,  (2)  an  uncertainty  in  the  maturation  of  technologies,  and  (3)  an  inability  to  compare 
the  impact  of  alternative  TRLs  on  the  system  as  a  whole  (Cundiff,  2003;  Dowling  &  Pardoe, 
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2005;  Mankins,  2002;  Meystel,  Albus,  Messina  &  Leedom,  2003;  Moorehouse,  2001;  Shishko, 
Ebbeler  &  Fox,  2003;  Smith,  2005;  Valerdi  &  Kohl,  2004). 

Based  on  these  fundamental  conjectures,  a  more  comprehensive  set  of  concerns  becomes 
relevant  when  the  TRL  is  amplified  from  the  level  of  an  individual  technology  to  a  system 
context  that  involves  the  interplay  of  multiple  technologies.  For  example,  in  NASA’s  Mars 
Climate  Orbiter,  the  failure  of  two — independently  evaluated — technologies  to  use  the  same 
units  (i.e.,  Metric  versus  English)  contributed  to  the  loss  of  the  spacecraft.  While  testing  is 
absolutely  necessary,  it  is  not  always  capable  of  catching  the  many  small  errors  that  can  occur 
when  two  different  components  of  software  and/or  hardware  exchange  data  in  a  raw  format.  If 
the  integration  of  two  pieces  of  technology  followed  some  sort  of  maturation  process,  just  as  the 
technology  itself  does,  this  would  provide  an  assessment  of  integration  readiness  and  a  direction 
for  improving  maturity  from  a  systems  context  during  the  development  process.  Not 
withstanding  the  previously  identified  limitations  of  the  TRE,  any  metric,  as  described  by 
Dowling  and  Pardoe  (2005),  should  not  lose  sight  of  what  makes  it  effective  and  efficient  in  an 
organization: 

1 .  The  way  the  value  is  used  should  be  clear. 

2.  The  data  to  be  collected  for  the  metric  should  be  easily  understood  and  easy  to  collect. 

3.  The  method  of  deriving  the  value  from  the  data  should  be  clear  and  as  simple  as  possible. 

4.  Those  for  whom  the  use  of  the  metric  implies  additional  cost  should  see  as  much  direct 
benefit  as  possible  (i.e.,  collecting  the  data  should  not  cost  more  than  its  value  to  the  de¬ 
cision  process). 

3.  Development  of  a  System  Readiness  Level 

In  theory,  technology  and  system  development  follow  similar  evolution  (or  maturation)  paths;  a 
technology  is  inserted  into  a  system  (e.g.,  evolutionary  acquisition)  based  on  its  maturity, 
functionality  and  environmental  readiness  and  ability  to  interoperate  with  the  intended  system. 
However,  many  of  the  factors  that  may  determine  the  successful  deployment  of  a  system  into  its 
operational  environment  are  not  always  effectively  implemented  during  the  developmental 
lifecycle  (Parsons,  2006).  Fundamentally,  any  system  under  development  is  composed  of  core 
technology  components  and  their  linkages  in  accordance  with  the  proposed  architecture. 
Henderson  and  Clark  (1990)  showed  that  the  distinction  between  the  relationships  of  the 
components  and  the  system  architecture  requires  two  types  of  knowledge:  component  knowledge 
and  architectural  knowledge  (i.e.,  knowledge  on  how  the  components  are  integrated).  These 
researchers  emphasized  that  systems  often  fail  because  attention  is  given  to  the  technology  while 
knowledge  of  the  linkages/integrations  is  overlooked.  They  explain  that  improper  attention  to 
the  linkages/integrations  has  an  impact  on  the  systems’  technical  evolution,  organizational 
experience,  recurrent  task,  and  technical  knowledge  as  they  relate  to  the  component  linkages.  It 
also  influences  the  product  architecture,  communication  channels,  and  problem  solving 
strategies.  Therefore,  while  the  TRL  provides  the  metric  for  describing  component  knowledge, 
based  on  Henderson  and  Clark,  one  would  still  be  interested  in  a  metric  that  provides  a 
description  of  architectural  knowledge  or  integration.  In  addition,  using  modeling  and 
simulation,  Ford  and  Dillard  (2008)  were  able  to  demonstrate  the  inherent  value  of  integration  to 
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the  success  of  evolutionary  acquisition.  They  were  able  to  demonstrate  the  relative  impact  of 
making  integrations  decisions  late  in  the  acquisition  lifecycle. 

While  there  have  been  some  efforts  to  develop  metrics  that  can  be  used  to  evaluate  integration 
(e.g.,  DoD,  1998,  March  30;  Mankins,  2002;  Fang,  Hu  &  Han,  2004;  Nilsson,  Nordhagen  & 
Oftedal,  1990),  there  is  a  need  for  a  metric  that  can  be  used  with  the  TRL  to  effectively 
determine  a  system  maturity.  This  paper  addresses  this  need  by  developing  a  system  maturity 
scale  that  incorporates  the  TRL  and  a  metric  of  integration  maturity,  which  is  described  below. 

3.1.  Integration  Readiness  Level 

The  application  of  ontology  metrics  to  support  integration  has  been  extensively  used  in  the 
computer  industry  to  define  the  coupling  of  components  (Orme,  Yao  &  Etzkom  2006;  2007),  but 
a  common  ontological  approach  to  technology  integration  for  system  development  has  been  far 
less  developed.  One  of  the  first  attempts  to  address  this  was  conducted  by  Mankins  (2002)  when 
he  proposed  an  Integrated  Technology  Analysis  Methodology  to  estimate  an  Integrated 
Technology  Index  (ITI).  The  ITI  was  then  used  for  a  comparative  ranking  of  competing 
advanced  systems.  The  study  brought  to  the  forefront  the  difficulty  of  progressing  through  the 
TRL  scale  and  choosing  between  competing  alternative  technologies.  It  did  not  adequately 
address  the  integration  aspects  of  systems  development.  Based  on  concerns  for  successful 
insertion  of  technologies  into  a  system,  the  Ministry  of  Defence  in  the  United  Kingdom 
developed  a  Technology  Insertion  Metric  that  includes,  among  other  things,  an  Integration 
Maturity  Level  (Dowling  &  Pardoe,  2005).  Building  upon  these  efforts,  Gove  (2007)  and  Gove, 
Sauser  and  Ramirez-Marquez  (2007)  performed  a  review  of  aerospace  and  defense-related 
literature  to  identify  the  requirements  for  developing  a  7-level  integration  metric  that  they  called 
Integration  Readiness  Level  (IRL).  These  factors  led  to  the  definition  of  the  requirements  for  an 
integration  metric,  which  are  to: 

1 .  Provide  an  integration-specific  metric,  to  determine  the  integration  maturity  between  two 
or  more  configuration  items,  components,  and/or  subsystems. 

2.  Provide  a  means  to  reduce  the  uncertainty  involved  in  maturing  and  integrating  a  tech¬ 
nology  into  a  system. 

3 .  Provide  the  ability  to  meet  system  requirements  during  the  integration  assessment  so  as  to 
reduce  the  integration  of  obsolete  technology  over  less  mature  technology. 

4.  Provide  a  common  platfonn  for  both  new  system  development  and  technology  insertion 
maturity  assessment. 

Using  these  requirements,  Gove  et  al.  (2007)  assessed  Mankin’s  Integrated  Technology  Index  ( 
2002),  Nilsson  et  al.’s  integration  metric  (1990),  Fang  et  al.’s  Interoperability  Assessment  Model 
(2004),  and  their  7-level  IRL  (Sauser  et  al.,  2006).  While  none  of  these  methods  met  all  the 
stated  requirements,  the  analysis  yielded  a  modified  9-level  IRL  which  did.  The  resulting  IRL  is 
a  systematic  analysis  of  the  interfacing  of  compatible  interactions  for  various  technologies  and 
the  consistent  comparison  of  the  maturity  between  integration  points  (i.e.,  TRLs)  and  is 
described  in  Table  1. 

Gove  et  al.  (2007)  also  evaluated  these  integration  maturity  metrics  with  multiple  system  case 
studies  (i.e.,  Mars  Climate  Orbiter,  Ariane  5,  two  Hubble  Space  Telescope  cases)  to  determine 
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how  effective  they  would  be  in  recognizing  integration  risks  in  development.  The  case  study 
analysis  showed  that  the  existing  approaches  to  integration  metrics  would  not  have  identified  the 
root  cause  of  the  development  risks.  Application  of  the  IRL  approach,  however,  was  shown  to 
have  highlighted  low  levels  of  integration  maturity  and  identified  specific  areas  of  development 
needing  further  management  and  engineering  attention.  Consequently,  we  use  this  IRL  in  the 
development  of  the  SRL. 


Table  1.  Integration  Readiness  Levels 

(Gove,  2007;  Gove  et  ah,  2007) 


IRL 

Definition 

Description 

9 

Integration  is  Mission  Proven 
through  successful  mission  opera¬ 
tions. 

IRL  9  represents  the  integrated  technologies  being  used  in  the 
system  environment  successfully.  In  order  for  a  technology  to 
move  to  the  TRL  9,  it  must  first  be  integrated  into  the  system 
and  then  proven  in  the  relevant  environment;  thus,  progressing 

IRL  to  9  also  implies  maturing  the  component  technology  to  the 
TRL  9. 

8 

Actual  integration  completed  and 
Mission  Qualified  through  test  and 
demonstration  in  the  system  envi¬ 
ronment. 

IRL  8  represents  not  only  the  integration-meeting  requirements, 
but  also  a  system-level  demonstration  in  the  relevant  environ¬ 
ment.  This  will  reveal  any  unknown  bugs/defects  that  could  not 
be  discovered  until  the  interaction  of  the  two  integrating  tech¬ 
nologies  was  observed  in  the  system  environment. 

7 

The  integration  of  technologies  has 
been  Verified  and  Validated  with 
sufficient  detail  to  be  actionable. 

IRL  7  represents  a  significant  step  beyond  IRL  6;  the  integration 
has  to  work  from  a  technical  perspective,  but  also  from  a  re¬ 
quirements  perspective.  IRL  7  represents  the  integration  meet¬ 
ing  requirements  such  as  performance,  throughput,  and  reliabil¬ 
ity. 

6 

The  integrating  technologies  can  Ac¬ 
cept,  Translate,  and  Structure  In¬ 
formation  for  its  intended  applica¬ 
tion. 

IRL  6  is  the  highest  technical  level  to  be  achieved;  it  includes  the 
ability  to  not  only  control  integration,  but  to  specify  what  infor¬ 
mation  to  exchange,  to  label  units  of  measure  to  specify  what  the 
information  is,  and  the  ability  to  translate  from  a  foreign  data 
structure  to  a  local  one. 

5 

There  is  sufficient  Control  between 
technologies  necessary  to  establish, 
manage,  and  terminate  the  integra¬ 
tion. 

IRL  5  simply  denotes  the  ability  of  one  or  more  of  the  integrat¬ 
ing  technologies  to  control  the  integration  itself;  this  includes  es¬ 
tablishing,  maintaining,  and  terminating. 

4 

There  is  sufficient  detail  in  the  Qual¬ 
ity  and  Assurance  of  the  integration 
between  technologies. 

Many  technology-integration  failures  never  progress  past  IRL  3, 
due  to  the  assumption  that  if  two  technologies  can  exchange  in¬ 
formation  successfully,  then  they  are  fully  integrated.  IRL  4 
goes  beyond  simple  data  exchange  and  requires  that  the  data  sent 
is  the  data  received  and  there  exists  a  mechanism  for  checking  it. 

3 

There  is  Compatibility  (i.e.,  com¬ 
mon  language)  between  technologies 
to  orderly  and  efficiently  integrate 
and  interact. 

IRL  3  represents  the  minimum  required  level  to  provide  success¬ 
ful  integration.  This  means  that  the  two  technologies  are  able  to 
not  only  influence  each  other,  but  also  to  communicate  interpret¬ 
able  data.  IRL  3  represents  the  first  tangible  step  in  the  maturity 
process. 

2 

There  is  some  level  of  specificity  to 
characterize  the  Interaction  (i.e., 
ability  to  influence)  between  tech¬ 
nologies  through  their  interface. 

Once  a  medium  has  been  defined,  a  “signaling”  method  must  be 
selected  such  that  two  integrating  technologies  are  able  to  influ¬ 
ence  each  other  over  that  medium.  Since  IRL  2  represents  the 
ability  of  two  technologies  to  influence  each  other  over  a  given 
medium,  this  represents  integration  proof-of-concept. 

1 

An  Interface  between  technologies 
has  been  identified  with  sufficient 

This  is  the  lowest  level  of  integration  readiness  and  describes  the 
selection  of  a  medium  for  integration. 
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detail  to  allow  characterization  of  the 
relationship. 


3.2.  System  Readiness  Level 

The  introduction  of  an  IRL  to  the  assessment  process  not  only  provides  a  check  as  to  where  the 
technology  is  on  an  integration  readiness  scale  but  also  presents  a  direction  for  improving 
integration  with  other  technologies.  Just  as  a  TRL  has  been  used  to  assess  the  risk  associated 
with  developing  technologies,  an  IRL  is  designed  to  assess  the  risk  associated  with  integrating 
these  technologies.  Now  that  both  the  technologies  and  integration  elements  can  be  assessed  and 
mapped  along  a  numerical  scale,  the  next  challenge  is  to  develop  a  metric  that  can  assess  the 
maturity  of  the  entire  system  that  is  under  development.  Sauser,  Ramirez-Marquez,  Henry  and 
DiMarzio  (2008)  were  able  to  demonstrate  how  the  TRLs  and  IRLs  for  any  system  under 
development  can  yield  a  measure  of  system  maturity  called  a  System  Readiness  Level  (SRL). 
The  rationale  behind  the  SRL  developed  by  Sauser  et  al.  (2008)  is  that  in  the  development 
lifecycle,  one  would  be  interested  in  addressing  the  following  considerations: 

■  Quantifying  how  a  specific  technology  is  being  integrated  with  every  other  technology  to 
develop  the  system. 

■  Providing  a  system-wide  measurement  of  readiness. 

The  computational  approach  for  the  SRL  has  been  considered  as  a  normalized  matrix  of  pair¬ 
wise  comparisons  of  the  TRLs  and  IRLs.  The  SRL  matrix  consists  of  one  element  for  each  of  the 
constituent  technologies  and,  from  an  integration  perspective,  quantifies  the  readiness  level  of  a 
specific  technology  with  respect  to  every  other  technology  in  the  system.  It  should  be  mentioned 
that  although  the  original  (1,9)  scale  for  both  the  TRL  and  IRL  can  be  used,  the  use  of 
normalized  values  allows  for  a  more  accurate  assessment  when  comparing  the  use  of  competing 
technologies.  Thus,  the  values  used  in  the  matrices  [TRL]  and  [IRL]  are  normalized  (0,1)  from 
the  original  (1,9)  levels  by  dividing  each  element  by  9. 

In  addition,  when  no  integration  is  present  between  two  technologies,  an  IRL  value  of  0  is 
assigned.  This  is  in  contrast  to  using  a  value  of  9  when  no  integration  is  present,  as  was 
originally  proposed  by  Sauser  et  al.  (2008).  Using  the  higher  value  of  9  gave  excessive  weight  to 
the  IRL  and  was  distorting  the  overall  SRL  value  upwards.  Consequently,  this  means  that  in  the 
future,  if  the  architecture  is  changed  such  that  those  two  technologies  become  integrated,  one  can 
go  back  and  apply  the  corresponding  IRL  value  of  that  new  integration  link.  For  integrations  to 
itself,  a  non-normalized  IRL  value  of  9  or  normalized  value  of  1  is  used.  The  reason  for  this  has 
a  philosophical  underpinning.  In  the  view  of  one’s  self,  it  is  a  matter  of  a  person  integrating 
various  parts  of  their  personality  into  a  harmonious,  intact  whole  with  the  purpose  of  keeping  the 
self  intact  and  uncorrupted.  For  this  reason,  when  interpreting  the  integration  of  a  technology  to 
itself,  we  define  it  as  uncorrupted  (i.e.,  fully  mature).  If  we  were  to  consider  the  integrations 
within  the  technology  independent  of  the  other  technologies,  then  we  would  be  calculating  a 
different  SRL  and,  thus,  be  considering  a  different  system  independent  of  the  system  of  interest. 
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3.3.  Calculating  the  SRL 

The  computation  of  the  SRL  is  a  function  of  the  TRL  and  IRL  matrices: 

■  Matrix  TRL  provides  a  blueprint  of  the  state  of  the  system  with  respect  to  the  readiness 
of  its  technologies.  TRL,  defined  as  a  vector  with  n  entries,  is  defined  in  Equation  1, 
where  TRL,  is  the  TRL  of  technology  i. 

TRL , 

TRL2 


■  Matrix  IRL  illustrates  how  the  different  technologies  are  integrated  with  each  other  from 
a  system  perspective.  For  a  system  with  n  technologies,  [IRL]  is  defined  in  Equation  2, 
where  1 R  L;/  is  the  IRL  between  technologies  i  and  j.  The  hypothetical  integration  of  a 
technology  i  to  itself  is  denoted  by  IRLu. 


(2) 


[«£], 


IRLU 

IRL\2  ■ 

..  IRLln 

IRL2\ 

irl22  . 

..  IRL2n 

IRLn, 

iRL„2  ■ 

••  iRLnn 

In  these  matrices,  the  standard  TRL  and  IRL  levels  corresponding  to  values  from  1  through  9 
should  be  normalized.  A  nonnalized  value  of  1  for  element  I R  L;/  can  be  understood  as  one  of  the 
following  with  respect  to  the  z'th  and  /th  technologies:  1)  they  are  completely  compatible  within 
the  total  system;  2)  they  do  not  interfere  with  each  other’s  functions;  3)  they  require  no 
modification  of  the  individual  technologies;  and  4)  they  require  no  further  integration  linkage 
development. 

In  any  system,  each  of  the  constituent  technologies  is  connected  to  a  minimum  of  one  other 
technology  through  a  bi-directional  integration.  The  way  each  technology  is  integrated  with 
other  technologies  is  used  to  formulate  an  equation  for  calculating  SRL.  This  SRL  equation 
consists  of  the  TRL  and  IRL  values  of  the  technologies  and  the  interactions  that  form  the  system. 
In  order  to  calculate  a  value  of  the  SRL  from  the  TRL  and  IRL  values,  we  propose  a  normalized 
matrix  of  pair-wise  comparison  of  the  TRL  and  IRL  values. 

Based  on  these  two  matrices,  an  SRL  matrix  is  acquired  by  obtaining  the  product  of  the  TRL  and 
IRL  matrices,  as  shown  in  Equation  3. 

(3)  \SRL]  =\IRL]  x\TRL 1  , 

The  SRL  matrix  consists  of  one  element  for  each  of  the  constituent  technologies  and,  from  an 
integration  perspective,  quantifies  the  readiness  level  of  a  specific  technology  with  respect  to 
every  other  technology  in  the  system  while  also  accounting  for  the  development  state  of  each 
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technology  through  the  TRL.  Mathematically,  for  a  system  with  n  technologies,  [SRL]  is  as 
shown  in  Equation  4. 


(4) 


SRLl 

IRLnTRL]  +  IRLl2TRL2  +  ...  +  IRLlnTRLn 

[SRL]  = 

srl2 

= 

IRL^TRJLy  +  IRL  22T RL  2 

SRL'- 

IRLnlTRLl  +  IRLn2TRL2  +  ...  +  IRLnnTRLn_ 

where  lRLij=IRLji. 


The  representation  of  each  of  the  SRL  values  obtained  in  Equation  4  addresses  the  first 
consideration  previously  discussed  in  Section  3.2.  Note  that  these  values  would  fall  within  the 
interval  (0,n);  so,  for  consistency,  for  each  technology,  say  i,  its  corresponding  SRL,  is  divided 
by  ni  ( ni  being  the  number  of  integrations  of  technology  i,  with  every  other  technology  as 
dictated  by  the  system  architecture — including  its  integration  to  itself)  to  obtain  its  normalized 
value  between  (0,1).  The  SRL  for  the  complete  system  is  the  average  of  all  such  normalized 
SRL  values,  as  shown  in  Equation  5.  Equal  weights  are  given  to  each  technology,  since  they  are 
each  identified  as  critical  technology  elements;  in  this  way,  a  simple  average  is  estimated.  A 
standard  deviation  can  also  be  calculated  to  indicate  the  variation  in  the  system  maturity  and 
parity  in  subsystem  development. 

SRL,  SRL ,  SRL , 

(5)  SRL  =  — ^ ^ 

n 

where  n,  is  the  number  of  integrations  with  technology  i  plus  its  integration  to  itself. 

The  SRL  metric  can  be  used  to  determine  the  maturity  of  a  system  and  its  status  within  a 
developmental  lifecycle.  Table  2  presents  an  example  of  how  the  various  levels  of  the  SRL  scale 
can  correlate  to  an  acquisition  lifecycle  (DoD,  2005a).  The  ranges  of  SRL  represented  in  Table  2 
are  derived  from  sensitivity  analysis  with  sample  systems.  While  we  are  working  to  verify  and 
validate  this  correlation  as  part  of  current  research,  we  contend  that  any  correlation  should  be 
accessed  based  unique  organizational  and  system  development  environments.  Also,  it  is 
important  to  note  that  in  this  correlation,  a  system  that  has  not  reached  full  maturity  is  capable  of 
transitioning  into  a  Production  phase.  This  is  predicated  on  the  reasoning  that  most  systems  are 
deployed  without  all  of  the  technologies  and  integrations  having  reached  full  maturity.  For 
example,  many  military  and  space  systems  cannot  be  verified  in  their  operational  environment 
until  deployed;  likewise,  many  systems  are  part  of  an  evolutionary  lifecycle  in  which  the  final 
maturity  will  be  verified  once  deployed  or  in  the  next  evolution. 
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Table  2.  System  Readiness  Levels 


SRL 

Acquisition  Phase 

Definitions  1 

0.90  to  1.00 

Operations  &  Support 

Execute  a  support  program  that  meets  operational  support 
performance  requirements  and  sustains  the  system  in  the  most 
cost-effective  manner  over  its  total  lifecycle. 

0.80  to  0.89 

Production 

Achieve  operational  capability  that  satisfies  mission  needs. 

0.60  to  0.79 

System  Development  & 
Demonstration 

Develop  system  capability  or  (increments  thereof);  reduce  in¬ 
tegration  and  manufacturing  risk;  ensure  operational  support- 
ability;  reduce  logistics  footprint;  implement  human  systems 
integration;  design  for  production;  ensure  affordability  and 
protection  of  critical  program  information;  and  demonstrate 
system  integration,  interoperability,  safety  and  utility. 

0.40  to  0.59 

Technology  Development 

Reduce  technology  risks  and  determine  appropriate  set  of 
technologies  to  integrate  into  a  full  system. 

0.10  to  0.39 

Concept  Refinement 

Refine  initial  concept;  develop  system/technology  strategy. 

NOTE:  These  ranges  have  been  derived  from  sensitivity  analysis  with  sample  systems.  They  are  currently 
undergoing  field  verification  and  validation  under  Naval  Postgraduate  School  Contract  #  N00244-08-0005. 


4.  Example  of  SRL  Calculation 

To  show  the  steps  and  analysis  involved  in  formulating  the  SRL,  the  following  example  will  use 
notional  data  (with  TRLs  that  range  from  a  low  of  6  to  a  high  of  9  and  IRLs  ranging  from  5  to  9) 
from  a  system  currently  under  development  for  a  family  of  surface  ships  in  the  US  Navy.  The 
system  architecture  analyzed  (see  Figure  1)  represents  an  end-to-end  integration  of  coimnand- 
and-control  capabilities  with  a  variety  of  unmanned  vehicles  and  intelligence,  surveillance,  and 
reconnaissance  sensor  packages.  These  elements  are  capable  of  autonomous  operations  and 
include  both  off-the-shelf  equipment  and  cutting-edge  new  development  networked  seamlessly 
together  to  enhance  effectiveness  and  efficiency.  For  this  system,  the  following  matrices  can  be 
created  for  the  TRL  and  1RL  (Equations  1  and  2). 
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(1) 

[TRL]20xl 


Figure  1.  Schematic  Architecture  of  System  X 
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As  indicated  in  the  above  integration  matrix,  we  assign  an  IRL  value  of  0  when  there  is  no 
integration  link  contemplated  between  any  two  technologies.  For  integration  to  itself,  an  IRL 
value  of  9  is  used.  Next,  we  normalize  the  [TRL]  and  [IRL]  matrices  by  dividing  each  element 
by  9.  Then,  we  calculate  [SRL]  as  follows  (Equation  3  and  4): 


(3  and  4) 


SRf 
SRL , 

SRL,, 


Table  3  indicates  the  calculated  values  for  each  SRL,. 


Table  3.  Individual  SRL  Values 


SRL, 

srl2 

srl3 

srl4 

SRLs 

SRLs 

srl7 

srl8 

SRL, 

SRL  io 

(0,  n.) 

2.000 

3.691 

2.605 

4.482 

1.963 

3.728 

2.000 

2.333 

2.000 

1.519 

(0,1) 

1.000 

0.923 

0.868 

0.640 

0.654 

0.746 

1.000 

0.778 

0.667 

0.759 

SRLn 

SRLi2 

srl13 

SRLi4 

srl15 

srl16 

SRL  17 

SRL  |g 

SRL  l9 

srl20 

(0,  no 

1.741 

1.556 

1.444 

1.333 

1.482 

1.568 

5.778 

2.358 

2.099 

2.210 

(0,1) 

0.580 

0.778 

0.722 

0.667 

0.741 

0.784 

0.722 

0.786 

0.699 

0.737 

The  calculated  Composite  SRL  scale  (Equation  5)  of  0.76  indicates  that  the  system  under 
development  should  be  in  the  System  Development  and  Demonstration  phase  (also  see  Figure  2). 
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SRLj  ]  SRL2  |  |  SRL„  SRL ,  |  SRL2  |  |  SRL20 

(5)  Composite  SRL  =  — ^ - — - — —  =  — - - - - —  =  0.76 

n  20 

Aside  from  the  SRL  providing  an  assessment  of  overall  system  development,  it  can  also  be  a 
guide  in  prioritizing  potential  areas  that  require  further  development.  That  is,  if  we  are 
considering  a  “systems-focused  approach”  to  our  methodology,  then  we  cannot  evaluate  a 
system  based  on  just  a  single  number,  such  as  the  Composite  SRL.  As  shown  in  our  example 
and  illustrated  by  Figure  2,  the  SRL,s  (technologies  with  their  integration  links  considered) 
present  a  spectrum  showing  some  subsystems  whose  readiness  levels  (i.e.,  SRL,)  are  in  the  three 
development  phases  other  than  the  Composite  SRL’s  System  Development  and  Demonstration 
phase.  While  it  could  be  argued  that  the  overall  SRL  is  only  as  good  as  the  lowest  SRL,,  this 
perspective  would  also  lose  sight  of  even  those  technologies  that  are  potentially  developing 
faster  than  the  system  (see  SRLi^j).  In  understanding  the  value  of  the  SRL  analysis,  we  must 
understand  the  spectrum  of  SRL,  and  its  relationship  to  the  Composite  SRL  (see  Figures  2  and 
4).  For  example,  the  value  of  considering  the  IRL  with  the  TRL  is  seen  in  Technology  1 1 .  This 
technology  has  a  TRL  of  9.  However,  when  we  consider  its  IRLs  (both  of  which  are  only  in 
level  5),  we  can  determine  it  not  only  is  less  mature  but  is  a  phase  behind  the  Composite  SRL. 
This  means  that  this  subsystem  (SRLn)  is  still  in  the  Technology  Development  phase,  while  the 
overall  system  is  already  in  the  System  Development  and  Demonstration  phase.  In  addition,  as 
shown  in  Figure  2,  20%  of  the  technologies  are  at  least  one  phase  ahead. 

Ideally,  this  type  of  analysis  can  facilitate  strategic  decisions  about  incremental  technology  and 
integration  investments  of  limited  resources.  For  example,  in  the  upcoming  budgetary  period  or 
fiscal  year,  resources  may  be  shifted  in  favor  of  accelerating  the  development  of  the  technologies 
and  integration  links  that  are  behind  and  temporarily  away  from  those  that  are  ahead — provided 
such  a  shift  is  technologically  and  organizationally  feasible.  This  capability  can  become 
important  when  a  specific  technology  is  a  conduit  for  downstream  technologies — its  maturity  is 
critical  for  the  system  to  reach  a  certain  level  of  maturity.  For  example,  the  system  diagram  in 
Figure  1  shows  that  Technology  4  is  such  a  technology.  If  the  systems  engineer  has  specified 
that  at  this  particular  time  period,  the  SRL  for  this  subsystem  must  be  at  least  0.80  before  the  rest 
of  the  technologies  can  be  developed  further,  the  program  manager  will  know  that  the  TRL  and 
IRL  for  Technology  4  have  to  be  improved  to  raise  its  SRL  from  the  current  value  of  0.64. 
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Figure  2.  SRL  Mapping  to  Defense  Acquisition 


5.  SRL  Relevance  and  Future  Research 


Given  the  ability  to  estimate  readiness  of  a  system  under  development  (summarized  in  Figure  3), 
organizations  can  systematically  evaluate  the  implications  of  using  alternative  technologies  or 
system  architectures,  prepare  development  plans  that  optimize  the  objectives  of  the  development 
team,  and  eventually  be  able  to  evaluate  and  monitor  the  progress  of  the  development  effort  to 
identify  problem  areas  and  corrective  measures  (example  in  Figure  4). 


Figure  3.  SRL  Methodology  and  Analysis  Flow 
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Figure  4.  Example  of  Documented  Status  via  Roll-up  Chart 

In  our  development  of  an  SRL  scale,  we  strived  to  maintain  a  systems-focused  approach  that 
would  create  a  metric(s)  to  address  some  of  the  current  concerns  with  the  TRL.  What  resulted 
was  a  set  of  metrics  and  an  approach  that  can  have  the  following  implications  on  defense 
acquisition: 

■  The  SRL,  IRL,  and  TRL  provide  an  enhanced  capability  alignment  through  the  identifica¬ 
tion  of  specific  technology,  integration,  and  system  maturities  that  can  be  used  as  a  trade- 
study  tool  to  select  the  most  appropriate  technologies  and  integrations  to  obtain  the  low¬ 
est  amount  of  risk,  cost,  and  time  and  satisfy  a  given  customer  need. 

■  The  SRL  [IRL,  TRL]  model  can  improve  customer  confidence  in  the  acquisition  manager 
by  providing  a  qualification  of  system  maturity  in  relation  to  system  functionality.  It  can 
also  provide  improved  understanding  of  the  system’s  mission  capabilities  in  tenns  of 
readiness  criteria. 

■  The  SRL  can  provide  an  assessment  of  maturity  at  multiple  architectural  layers.  Any 
single  SRL  assessment  contains  multiple  SRL  assessments  from  the  SRL  vector,  which 
can  provide  insight  into  the  interdependencies  of  different  sub-functions  and  how  they  fit 
within  the  larger  architecture. 

■  The  SRL  can  provide  a  fast,  iterative  assessment  that  can  be  repeated  and  traced  during 
development.  This  can  facilitate  a  valuable  exercise  in  architecture  examination  and  crea¬ 
tion,  which  can  allow  for  better  system  understanding  and  (re)formation. 
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■  The  SRL  and  IRL  allow  for  other  factors  (in  addition  to  technology  readiness)  as  meas¬ 
ures  of  maturity.  In  addition,  decision-makers  can  consider  factors  such  as  obsolescing — 
by  comparative  analysis  of  multiple  technologies  to  acquisition — and  the  optimization  of 
technology  maturation  investment  and  transition  funding.  This  is  currently  an  area  of  fu¬ 
ture  research. 

■  The  SRL,  IRL,  and  TRL  provide  common  ontology  to  measure  and  describe  acquisition 
development,  system  development  and  technology-insertion  evaluation. 

■  The  IRL  reduces  the  uncertainty  involved  in  integrating  a  technology  into  a  system  and 
identifies  integration  as  a  separate,  specific  metric. 

Despite  the  utility  of  the  SRL,  it  is  not  without  a  core  limitation.  That  is,  our  tactical  approach  to 
the  SRL  was  similar  to  that  of  calculating  a  student’s  grade  point  average  (GPA) — in  which 
ordinal  data  is  given  numeric  value  in  order  to  assess  overall  progression  or  performance.  This 
approach  also  incurs  a  key  limitation  to  assessing  a  system’s  development.  Accordingly,  the 
SRL  for  one  system  cannot  be  compared  to  the  SRL  of  another  system  unless  they  are  the  same 
system.  For  example,  it  is  difficult  to  compare  a  student  with  a  3.2  GPA  (on  a  4.0  scale)  in 
physics  with  a  student  that  has  a  3.8  GPA  in  biology.  These  students  belong  to  different  systems 
of  education,  but  they  are  evaluated  with  the  same  system  of  metrics.  Likewise,  the  SRL  can  be 
effective  for  assessing  the  progressive  maturity  of  the  system  of  interest,  but  it  is  questionable  to 
compare  the  maturity  progression  of  two  systems  against  each  other  because  of  other  inherent 
factors  related  to  the  context  in  which  the  system  is  being  developed. 

Further  trials  using  real  case  studies  are  necessary  in  order  to  verify  the  formulation  of  the  SRL, 
as  well  as  to  establish  its  validity.  These  will  also  be  necessary  in  order  to  illustrate  the  benefits 
of  SRL  in  tenns  of  improved  risk  management  and  value  added  at  key  decision  points  along  the 
acquisition  lifecycle.  When  the  validity  of  the  SRL  is  established,  it  can  then  be  expanded  to 
incorporate,  where  necessary,  other  measures  of  readiness,  such  as  Manufacturing  Readiness 
Level  (MRL).  As  with  any  research,  the  fundamental  objective  is  to  increase  our  understanding 
by  asking  questions  that  lead  to  more  questions.  Thus,  for  future  research  in  system  maturity 
assessment  and  defense  acquisition,  we  propose  some  of  the  following  questions: 

■  Are  there  variations  in  how  system  maturity  assessment  is  used  with  various  lifecycles, 
e.g.,  linear  acquisition,  evolutionary  acquisition,  revolutionary  acquisition? 

■  What  are  the  implications  of  system  maturity  levels  for  the  integration  of  open  systems 
into  evolutionary  acquisition? 

■  What  are  the  impacts  of  disruptive  technologies  on  systems  maturity  forecasting? 

■  How  does  vendor  selection  impact  system  maturity  assessment? 

■  How  do  other  maturity  metrics,  such  as  the  Manufacturing  Readiness  Level  (MRL),  work 
with  the  IRL  and  SRL? 
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■  How  can  the  techniques  of  system  maturity  assessment  be  used  for  trade-off  analysis  of 
competing  technologies  or  systems? 

■  What  are  the  impacts  of  obsolescence  to  system  maturity  planning  and  road  mapping? 

■  What  are  the  single-technology  refreshment  optimization  considerations  for  asynchro¬ 
nous  refreshment  frequency? 

■  What  are  the  multi-objective  optimization  considerations  for  asynchronous  refreshment 
frequency? 

■  What  are  the  uncertainties  surrounding  the  lifecycle  curve  for  system  maturity? 

■  How  can  we  consider  the  environmental  costs  throughout  a  system’s  lifecycle? 

6.  Conclusions 

We  contend  that  the  IRL  is  necessary  because  in  some  programs,  integration  elements  have  been 
overlooked  and  have  resulted  in  major  debacles.  We  also  introduced  the  development  of  a 
system-focused  approach  for  managing  system  development  and  for  making  effective  and 
efficient  decisions  during  the  defense  acquisition  process.  To  accomplish  this,  we  developed  a 
SRL  scale  incorporating  both  the  current  TRL  and  the  proposed  IRL  scale.  We  then  described 
the  foundations  of  the  SRL  and  demonstrated  the  techniques  for  determining  current  readiness  of 
a  system  to  determine  its  position  in  the  defense  acquisition  lifecycle.  We  summarized  our 
approach  (describing  how  it  may  be  used  within  defense  acquisition),  showed  a  specific  example 
of  how  the  analysis  could  be  reported,  and  provided  some  questions  for  future  research. 

The  DoD  Technology  Readiness  Assessment  (TRA)  states  that  “the  TRA  should  not  be  the  sole 
means  of  discovering  technology  risk”  (DoD,  2005b).  Furthermore,  as  stated  earlier,  the  GAO 
has  reported  that  the  DoD  needs  additional  metrics  for  evaluating  weapons  systems.  While 
metrics  can  identify  critical  parameters,  establish  milestones  to  assess  progress,  provide  direction 
for  risk  management/mitigation,  or  sustain  entry  and  exit  criteria  for  major  milestones,  we  must 
keep  in  mind  the  four  guidelines  for  effective  and  efficient  metrics  by  Dowling  and  Pardoe 
(2005)  as  described  earlier.  Accordingly,  we  attempted  to  follow  these  guidelines  and  proposed 
the  inclusion  of  a  separate  maturity  scale  to  measure  the  progress  of  the  development  of  the 
integration  links  of  a  system  and  the  system  as  a  whole. 

We  consider  the  TRL  to  be  simple  and  understandable;  however,  some  ambiguity  exists,  in  part 
due  to  the  extrapolation  of  the  TRL  beyond  what  it  was  intended  to  do.  We  believe  that  the  IRL 
mimics  the  value  of  the  TRL  in  that  it  is  simple  and  understandable,  but  we  contend  that  the 
interpretation  of  the  individual  IRL  levels  may  need  more  clarification  before  the  IRL  can 
become  a  metric  in  practice.  The  combination  of  the  TRL  and  IRL  for  the  formulation  of  the 
SRL  was  not  a  simple  endeavor,  as  many  alternative  mathematical  approaches  were  pursued 
(Sauser  et  al.,  2007).  The  chosen  approach  was  used  because  it  was  the  simplest  and  most  robust 
with  respects  to  its  sensitivity  to  changes  in  any  TRL  or  IRL  within  a  system.  While  the  addition 
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of  any  metric  means  incurring  additional  costs  for  an  organization,  we  consider  the  addition  of 
the  IRL  and  SRL  as  a  cost  savings,  as  they  are  able  to  identify  factors  that  have  been  significant 
failures  in  many  system-development  programs.  Finally,  we  attempt  to  focus  the  development  of 
these  metrics  based  on  data  that  would  normally  be  available  to  any  systems  engineer  (e.g., 
system  architectures,  baselines).  Even  with  what  we  consider  to  be  a  valuable  contribution  to  the 
assessment  of  system  maturity,  the  additive  value  of  “readiness”  metrics  carries  with  it  the 
additive  drawbacks:  (a)  Subjectivity  and  Human-intensiveness — human-intensive  assessments 
can  be  overly  optimistic  and  contain  inherent  variation  or  ambiguity  that  is  averaged  away  and 
which  some  of  the  existing  approaches  may  fail  to  prevent;  and  (b)  Limited  Focus — while  this  is 
not  the  intent,  focusing  on  single  or  a  limited  subset  of  numbers  can  draw  attention  away  from 
other  core  issues. 

In  conclusion,  the  conceptual  development  of  these  or  any  metrics  and  tools  have  outpaced  their 
validation  and  verification  in  the  field.  What  is  necessary  now  is  to  have  greater  involvement 
from  practitioners  so  the  acquisition  community  can  agree  to  a  common  measurement  and 
language  that  can  only  improve  the  system  development  and  acquisition  process. 
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